This PR adds support for using a custom BIO method for performing SSL/TLS I/O. It is an alternative to #736 and a possible fix for #731.

Summary

Currently, the openssl gem uses a socket BIO (over non-blocking sockets) that bypasses the Ruby I/O layer, except for calling io_wait_readable/io_wait_writable to wait for I/O readiness. This prevents or makes it difficult to do encrypted I/O over alternative transports such as proxy connections (#731) or virtual sockets, for example in a testing situation.

I've also been looking at providing a better way to integrate the openssl gem with a fiber scheduler and specifically being able to use it in conjunction with a low-level API for performing I/O using io_uring that I'm developing.

The aim of this PR is to provide a minimal API that allows setting a custom BIO method that either uses the stock IO#read and IO#write methods to perform I/O, or alternatively use custom procs to perform read and write operations, which will allow complete freedom for performing I/O using a low-level API, a proxy connection or any virtual interface.

The proposed solution is based on the following design principles:

• Keep the existing I/O implementation as the default behavior.
• Minimize API changes.
• Don't touch the implementation of the different SSLSocket I/O methods in ossl_ssl.c: #read, #write, #connect, #accept etc.
• Minimize copying of buffers.
• Maximize flexibility for custom BIO methods.

API

We add two methods:

• SSLSocket#bio_method: get the socket's BIO method.

• SSLSocket#bio_method=: set the socket's BIO method.

The setter method accepts the following:

• nil: use the default socket BIO

• IO: use the given IO instance to perform I/O, via its #read and #write methods. This will usually be the underlying socket object. Example usage:

ssl.bio_method = ssl.to_io

• Object: use the given object to perform I/O, using the same interface as for an IO. Example usage:

class MyIO
  def read(len)
    'foo'
  end

  def write(str)
    str.bytesize
  end
end

ssl.bio_method = MyIO.new

• [read_proc, write_proc]: use the given pair of read/write procs to perform IO.

The read proc takes an IO::Buffer and a maximum read length, and should return the number of bytes read. The write proc takes an IO::Buffer and a write length, and should return the number of bytes written. Example usage:

 io = ssl.to_io
 ssl.bio_method = [
   ->(buf, maxlen) {
     str = io.read(maxlen)
     len = str.bytesize
     buf.set_string(str)
     len
   },
   ->(buf, len) {
     str = buf.get_string(0, len)
     io.write(str)
   }
 ]

Implementation

Since we're just changing the BIO associated with the SSLSocket instance, we don't need to touch the I/O implementation in functions such as ossl_ssl_read_internal. The custom BIO interface will never return SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, and thus the I/O operation will complete immediately after the call to e.g. SSL_read.

Since the custom BIO read and write hooks receive raw char * buffers, we need to pass the buffer as either String (in the case of IO/Object method), or IO::Buffer (in the case of read/write procs). The advantage of using a IO::Buffer is that there's no need to copy data between the raw buffer and the string (or vice versa). Hopefully, in the future, IO#read and IO#write would be able to accept an IO::Buffer as well as String as buffer.

Performance

A preliminary benchmark (source) shows a significant advantage to using a custom BIO method. This benchmarks measures the performance of the default socket BIO, the IO custom BIO method, and a custom BIO method using the UringMachine low-level API.

ruby 4.0.1 (2026-01-13 revision e04267a14b) +YJIT +PRISM [x86_64-linux]
Warming up --------------------------------------
             default     2.108k i/100ms
             BIO: IO     2.925k i/100ms
             BIO: UM     2.801k i/100ms
Calculating -------------------------------------
             default     26.535k (± 2.8%) i/s   (37.69 μs/i) -    132.804k in   5.009171s
             BIO: IO     31.038k (± 5.2%) i/s   (32.22 μs/i) -    155.025k in   5.009282s
             BIO: UM     27.835k (± 5.7%) i/s   (35.93 μs/i) -    140.050k in   5.050643s

Comparison:
             default:    26534.7 i/s
             BIO: IO:    31038.5 i/s - 1.17x  faster
             BIO: UM:    27834.9 i/s - same-ish: difference falls within error

Of course, the performance implications need to be investigated more thoroughly and may vary by OS, OpenSSL version, machine and network setup etc, and also concurrency.

OpenSSL and Ruby Compatibility

The implementation depends on the availability of BIO_meth_new and associated functions, which were added in OpenSSL 1.1.0.

The implementation also depends on the availability of the IO::Buffer c API, namely rb_io_buffer_new (available since Ruby 3.1) and rb_io_buffer_free_locked (available since Ruby 3.3).

Future work

• More tests, more benchmarks.
• Add bio_method kwarg to SSLSocket.new/SSLSocket.open (see also Add sync_close kwarg to SSLSocket.new #996).
• Eventually, if no problems are encountered, set the default BIO method to use the underlying socket and its #read and #write methods.

cc @HoneyryderChuck @ioquatix @rhenium